Image processing apparatus, image processing method and program

ABSTRACT

According to some aspects, an image processing apparatus is provided, comprising circuitry configured to receive an input image, the input image being supplied as a stereoscopic image including a left image for a left eye and a right image for a right eye, calculate depth information for each of a plurality of sub-regions of the input image based at least in part on the right image and the left image, and determine, for each of the plurality of sub-regions of the input image, at least one luminance component based at least in part on the depth information and a function indicating a relationship between depth information and luminance value.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation of and claims the benefit under 35U.S.C. §120 of U.S. patent application Ser. No. 14/150,912, titled“IMAGE PROCESSING APPRATUS, IMAGE PROCESSING METHOD AND PROGRAM,” filedon Jan. 9, 2014, which is a continuation of U.S. patent application Ser.No. 12/951,116, titled “IMAGE PROCESSING APPARATUS, IMAGE PROCESSINGMETHOD AND PROGRAM,” filed on Nov. 22, 2010, now U.S. Pat. No.8,660,337, issued Feb. 25, 2014, which claims priority under 35 U.S.C.§119(a) to Japanese Patent Application JP 2009-270077, filed on Nov. 27,2009. The entire contents of these applications are hereby incorporatedby reference in their entireties.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, an imageprocessing method and a program, and particularly relates to an imageprocessing apparatus, an image processing method and a program capableof providing 3D images with less sense of discomfort and uncomfortablefeeling for human beings by adjusting parameters of monocular 3Dinformation based on visual sense characteristics of human beings.

2. Description of the Related Art

As a method of displaying a 2D image on a 2D display and allowing aviewer to perceive the 2D image as a 3D image, there exists a method ofpresenting a 2D image for a left eye and a 2D image for a right eye inwhich binocular parallax (parallax between the left eye and the righteye) to the left and right eyes of the viewer respectively.

As information for allowing a human to perceive stereoscopic effect anddepth of an object, there are binocular 3D information by both eyes andmonocular 3D information by a single eye. A human perceives stereoscopiceffect and depth of the object or space by combining the monocular 3Dinformation and the binocular 3D information. As the binocular 3Dinformation, for example, binocular parallax, horizontal convergence andthe like can be cited. As the monocular 3D information, for example,shading, contrast, color, a spatial frequency, shielding relation andthe like can be cited.

When stereoscopic effect and depth are desired to be enhanced in thecase of displaying the 2D image on the 2D display and allowing theviewer to perceive the 2D image as the 3D image, for example, a methodof increasing binocular parallax which is one of the binocular 3Dinformation can be considered.

However, to increase the binocular parallax has the following problemsin the light of an ocular structure and visual sense characteristics ofhuman beings. That is, human eyeballs are normally in a parallel or arather inward convergence state, therefore, when the binocular parallaxis increased more than the distance between pupils, the eyeballs are ina divergence state in which both eyes are directed to the outside. Thedistance between pupils differs depending on the age or sex, therefore,a person having distance between pupils which is smaller than the normaldistance is liable to be in the divergence state.

In the real world, sight lines of both eyes are directed to a gaze pointas well as the focus of eyes is achieved on the point, therefore,distance of convergence of eyeballs corresponds to distance ofadjustment thereof. However, when allowing the viewer to perceive the 3Dimage by the 2D image for the left eye and the 2D image for the righteye, the convergence can be adjusted on a position perceived as the 3Dimage, while the adjustment is focused on an image display surface,therefore, the distance by convergence of eyeballs does not correspondsto the distance by adjustment thereof. Accordingly, to emphasisstereoscopic effect and depth by increasing binocular parallax changesthe distance by the convergence of eyeballs and the distance by theadjustment to a direction not correspond to each other, which may allowthe viewer to perceive artificiality or to feel discomfort and visualfatigue.

In order to reduce uncomfortable feeling and visual fatigue, a method ofadjusting the binocular parallax is proposed. For example, in a methodproposed in Japanese Patent No. 3749227 (Patent Document 1), pluralsample images in which binocular parallax is set to different values arepresented and whether the presented images are permitted or not isallowed to be responded to adjust the binocular parallax.

However, when uncomfortable feeling and visual fatigue of the viewer areintended to be reduced, the binocular parallax is basically adjusted toa direction of reducing the stereoscopic effect and depth, therefore,realistic sensation and reality are reduced. Additionally, whenstereoscopic effect and depth perceived by binocular 3D information aredifferent from stereoscopic effect and depth perceived from monocular 3Dinformation, the viewer may feel artificiality.

Therefore, it is not preferable that stereoscopic effect and depth senseof the 3D image are enhanced by increasing the binocular parallax.

On the other hand, a method of enhancing stereoscopic effect and depthsense by using monocular 3D information is also proposed. For example,in JP-A-2001-238231 (Patent Document 2), a method of changingcharacteristics of shading, shielding relation and a blurring stateaccording to a depth position of an object in an image to enhance thedepth sense is proposed.

SUMMARY OF THE INVENTION

However, in Patent Document 2, which parameter should be set to whichvalue based on which calculation equation is not specifically disclosed.Even when the value is set by trial and error, it is not certified thatan obtained 2D image for the left eye and a 2D image for the right eyeare natural and comfortable for human beings, and they may rather theviewer to feel artificiality or uncomfortable feeling or they may causevisual fatigue.

In view of the above, it is desirable to provide 3D images with lessersense of discomfort and uncomfortable feeling for human beings byadjusting parameters of monocular 3D information based on visual sensecharacteristics of human beings.

According to one embodiment of the invention, there is provided an imageprocessing apparatus including a depth information extraction means forextracting depth information from an input 3D image, a luminanceextraction means for extracting luminance components of the 3D image, acontrast extraction means for extracting contrast components of the 3Dimage based on the luminance components of the 3D image extracted by theluminance extraction means, a storage means for storing a performancefunction indicating relation between the contrast components of the 3Dimage and depth amounts subjectively perceived, which is determinedbased on visual sense characteristics of human beings and a contrastadjustment means for calculating present depth amounts of the inputted3D image from the contrast components of the 3D image extracted by thecontrast extraction means based on the performance function with respectto at least one of a near side region and a deep side region of theinputted 3D image which are determined from the depth informationextracted by the depth information extraction means and adjustingcontrast components of the inputted 3D image based on the calculatedpresent depth amounts and a set depth adjustment amount.

According to one embodiment of the invention, there is provided an imageprocessing method of an image processing apparatus storing a performancefunction indicating relation between contrast components of the 3D imageand depth amounts subjectively perceived, which is determined based onvisual sense characteristics of human beings and performing adjustmentof depth sense of the inputted 3D image, which includes the steps ofextracting depth information from the 3D image, extraction luminancecomponents of the 3D image, extraction of contrast components of the 3Dimage based on the extracted luminance components of the 3D image,calculating present depth amounts of the inputted 3D image from thecontrast components of the extracted 3D image based on the performancefunction with respect to at least one of a near side region and a deepside region of the inputted 3D image which are determined from theextracted depth information and adjusting contrast components of theinputted 3D image based on the calculated present depth amounts and theset depth adjustment amount.

According to one embodiment of the invention, there is provided aprogram allowing a computer to execute processing of extracting depthinformation from an inputted 3D image, extracting luminance componentsof the 3D image, extracting contrast components of the 3D image based onthe extracted luminance components of the 3D image, calculating presentdepth amounts of the inputted 3D image which are subjectively perceivedfrom the extracted contrast components of the 3D image based on aperformance function indicating relation between contrast components ofthe 3D image and the depth amounts subjectively perceived, which isdetermined based on visual sense characteristics of human beings withrespect to at least one of a near side region and a deep side region ofthe inputted 3D image which are determined from the extracted depthinformation, and adjusting contrast components of the inputted 3D imagebased on the calculated present depth amounts and a set depth adjustmentamount.

According to the embodiments of the invention, depth information isextracted from the inputted 3D image, luminance components of the 3Dimage are extracted, contrast components of the 3D image are extractedbased on the extracted luminance components of the 3D image, presentdepth amounts of the inputted 3D image which are subjectively perceivedare calculated from the extracted contrast components of the 3D imagebased on the performance function indicating relation between contrastcomponents of the 3D image and the depth amounts subjectively perceived,which is determined based on visual sense characteristics of humanbeings with respect to at least one of a near side region and a deepside region of the inputted 3D image which are determined from theextracted depth information, and contrast components of the inputted 3Dimage are adjusted based on the calculated present depth amounts and theset depth adjustment amount.

The image processing apparatus may be an independent apparatus as wellas an internal block forming one apparatus.

According to the embodiments of the invention, the depth sense of the 3Dimage can be enhanced.

Also according to the embodiments of the invention, parameters ofmonocular 3D information are adjusted based on visual sensecharacteristics of human beings, thereby providing the 3D image withlesser sense of discomfort or uncomfortable feeling for human beings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a configuration example of an imageprocessing apparatus according to a first embodiment of the invention;

FIG. 2 shows performance functions obtained by quantifying relationbetween Michelson contrasts C obtained by visual experiments and depthamounts D substantially perceived by contrast;

FIG. 3 is a flowchart for explaining contrast adjustment processingperformed by the image processing apparatus of FIG. 1;

FIG. 4 shows an example of a 3D image when supplied in the first dataformat;

FIG. 5 shows an example of a depth image obtained by visualizing depthinformation extracted from the 3D image;

FIG. 6 shows an example of a luminance image generated by extractingluminance components;

FIG. 7 shows an example of spatial frequency component images obtainedby visualizing the extraction results obtained by extracting respectivespatial frequency components;

FIG. 8 is a block diagram showing a configuration example of an imageprocessing apparatus according to a second embodiment of the invention;

FIG. 9 is a flowchart for explaining contrast adjustment processingperformed by the information processing apparatus of FIG. 8; and

FIG. 10 is a block diagram showing a configuration example of a computeraccording to an embodiment of the invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, modes for carrying out the invention (referred to asembodiments in the following description) will be explained. Theexplanation is made in the following order.

1. First Embodiment (an embodiment of changing depth sense by adjustingcontrast)

2. Second Embodiment (an embodiment capable of adjusting contrast whilereflecting user's preference)

1. First Embodiment Configuration Block Diagram of an Image ProcessingApparatus

FIG. 1 shows a configuration example of an image processing apparatusaccording to a first embodiment of the invention.

An image processing apparatus 1 includes an image input unit 11, a depthinformation extraction unit 12, a luminance extraction unit 13, aspatial frequency extraction unit 14, a contrast extraction unit 15, amemory unit 16, a contrast adjustment unit 17, an image processing unit18 and an image output unit 19.

The image processing apparatus 1 performs processing of changing(adjusting) contrast of a 3D image with respect to the inputted 3D imagefor enhancing depth sense of the 3D image. Here, the 3D image indicatesa 2D image for allowing a viewer to perceive the 3D image.

The image input unit 11 receives the 3D image from the outside andsupplies the inputted 3D image to the depth information extraction unit12 and the luminance extraction unit 13. There are various types of dataformats in the 3D image inputted from the outside, however, the dataformat does not matter. As data formats for the 3D image, for example,there are a first data format in which the image is supplied as astereoscopic image including an image for a left eye and an image for aright eye, a second data format in which the image is supplied as aformat of a multi-viewpoint image including three or more pluralviewpoint images, a third data format in which the image is supplied asa format of a 2D image and depth information thereof and so on. In thefollowing description, when a word of “an image” or “a luminance image”is used as a processing target, the word means respective images for theleft eye and the right eye in the first data format, respective pluralviewpoint images in the second data format and the 2D image suppliedwith depth information in the third data format.

The depth information extraction unit 12 extracts depth information fromthe 3D image supplied from the image input unit 11. Specifically, whenthe 3D image is inputted in the first data format, the depth informationextraction unit 12 calculates pixel misalignment of corresponding pointsin stereoscopic images, namely, binocular parallax, and approximatelycalculates depth information based on the calculated binocularinformation. The binocular parallax can be calculated by using methodssuch as a block matching method and a DP matching method.

When the 3D image is inputted in the second data format, the depthinformation extraction unit 12 calculates binocular parallax withrespect to corresponding two viewpoint images in three or more viewpointimages and approximately calculates depth information from thecalculated binocular parallax.

Furthermore, when the 3D image is inputted in the third data format, thedepth information extraction unit 12 extracts the supplied depthinformation as the depth information is directly supplied.

The depth information extraction unit 12 supplies the depth informationextracted from the 3D image to the contrast adjustment unit 17. Thedepth information is used as information for specifying a processingregion in which contrast is adjusted.

In the present embodiment, the 3D image as the processing target fromwhich depth information is extracted is supplied from the image inputunit 11 to the depth information extraction unit 12 as described above,however, it is also preferable that the 3D image as a processing resultof the luminance extraction unit 13 is supplied to the depth informationextraction unit 12.

The luminance extraction unit 13 extracts luminance components of the 3Dimage supplied from the image input unit 11. For example, when thesupplied image is represented by linear RGB values in the RGB colorsystem, the luminance extraction unit 13 converts the values into aluminance value Y by the following equation (1) prescribed by ITU-RBT709 to thereby extract luminance components of the 3D image.

Y=0.2126R+0.7152G+0.0722B   (1)

The image formed by luminance values Y of respective pixels is referredto as a luminance image. It is not always necessary that the image isrepresented by the format including RGB values (RGB signals), and whenthe image is represented by XYZ values in the CIE XYZ color system, theimage formed by luminance values Y is referred to as the luminanceimage. The luminance value can be calculated (extracted) by methodsother than the method of using the equation (1).

The spatial frequency extraction unit 14 extracts given spatialfrequency components of the 3D image. For the extraction of spatialfrequency components, for example, a method of using a Gabor filter canbe applied. The Gabor filter approximates signal responsecharacteristics in the visual system, and a function of the filter g(x,y, λ, θ, Ψ, σ and γ) is represented by an equation (2).

$\begin{matrix}{{g\left( {x,y,\lambda,\theta,\psi,\sigma,\gamma} \right)} = {{\exp \left( {- \frac{\left( {{x\; \cos \; \theta} + {y\; \sin \; \theta}} \right)^{2} + {\gamma^{2}\left( {{{- x}\; \sin \; \theta} + {y\; \cos \; \theta}} \right)}^{2}}{2\sigma^{2}}} \right)}{\cos \left( {{2\pi \frac{{x\; \cos \; \theta} + {y\; \sin \; \theta}}{\lambda}} + \psi} \right)}}} & (2)\end{matrix}$

In the equation (2), (x, y) represent coordinate values of a luminanceimage, λ represents a wavelength corresponding to a spatial frequency, θrepresents an orientation (direction), Ψ represents a phase, σrepresents dispersion of Gaussian distribution, and γ represents anaspect ratio, respectively. The spatial frequency in this case isdefined by tint variation (contrast) of white and black entering into aviewing angle of 1 degree of human eyes, and a unit thereof is “cpd(cycle per degree)”.

For example, the spatial frequency extraction unit 14 extracts a regionhaving the spatial frequency component of 1 cpd in the luminance imageby convoluting the Gabor filter function g(x, y, λ, θ, Ψ, σ and γ) ofthe equation (2) in which λ is the wavelength of 1 cpd with theluminance values Y of the luminance image extracted by the luminanceextraction unit 13. When spatial frequency components of 2 cpd and 4 cpdin the luminance image are extracted, regions can be calculated byconvoluting the Gabor filter function g(x, y, λ, θ, Ψ, σ and γ) in whichλ is the wavelength of 2 cpd or 4 cpd with the luminance values Y of theluminance image.

That is, it is found that which spatial frequency component is includedin which region of the luminance image by convoluting the Gabor filterfunction g(x, y, λ, θ, Ψ, σ and γ) of the equation (2) in which λ is agiven wavelength of the spatial frequency with the luminance values Y ofthe luminance image extracted by the luminance extraction unit 13.

As the method of extracting the region having a given spatial frequencycomponent of the luminance image, other methods other than the above canbe applied as long as which component of the spatial frequency isincluded in which region of the luminance image can be found by themethod. For example, the spatial frequency component may be extracted byusing Fourier transform and the like. However, the spatial frequencycomponents of the whole (region) of the luminance image to betransformed can be obtained in Fourier transform, therefore, it isdifficult to extract the spatial frequency components in respectiveregions unless Fourier transform is performed by dividing the luminanceimage into given regions. On the other hand, it is found which spatialfrequency component is included in which region all at once with respectto the whole luminance image to be processed when using the Gaborfilter.

The processing of extracting the spatial frequency by the spatialfrequency extraction unit 14 is necessary when a performance function tobe applied in the contrast adjustment unit 17 is changed according togiven spatial frequencies. Therefore, when the same performance functionis applied to the whole luminance image in the contrast adjustment unit17, the spatial frequency extraction unit 14 can be omitted.

The contrast extraction unit 15 extracts contrast components of the 3Dimage by using the luminance component of the 3D image extracted in theluminance extraction unit 13.

Specifically, the contrast extraction unit 15 determines a region of n×mpixel (n, m≧2) in horizontal×vertical as a unit for processing ofextracting the contrast components. Then, the contrast extraction unit15 moves the processing unit region from the upper left end of theluminance image in the Raster scan direction by the given number ofpixels to thereby calculate Michelson contrasts C of plural processingunit regions.

The Michelson contrast C is defined by an equation (3).

$\begin{matrix}{C = \frac{L_{\max} - L_{\min}}{L_{\max} + L_{\min}}} & (3)\end{matrix}$

In the equation (3), L_(max) represents the maximum value of theluminance values Y in the processing unit region and L_(min) representsthe minimum value of the luminance values Y in the processing unitregion.

The size of the processing unit region is not particularly limited,however, the optimum size can be determined from the relation betweenthe viewing angle and the number of pixels.

The movement amount (number of pixels) at the time of moving theprocessing unit region is not also particularly limited and can beoptionally determined. Respective processing unit regions may be set soas to be overlapped at parts thereof or may be set in tiled patterns soas not to be overlapped. In the case of giving weight on the accuracy,that is, in the case of adjusting contrast to obtain smoother images,the processing unit region may be moved pixel by pixel.

The contrast extraction unit 15 supplies the Michelson contrasts C ofrespective processing unit regions calculated as contrast components ofthe 3D image to the contrast adjustment unit 17 with the luminanceimage.

The memory unit 16 stores performance functions obtained by quantifyingthe relation between the Michelson contrasts C and depth amounts D (alsoreferred to as subjective depth amounts D) perceived subjectively by thecontrast which have been obtained by visual experiments.

Relation Between Michelson Contrasts C and Subjective Depth Amounts D

FIG. 2 shows performance functions obtained by quantifying the relationbetween the Michelson contrasts C obtained by visual experiments and thesubjective depth amounts D obtained at that time, which are plotted witha logarithm axis of Michelson contrasts as the horizontal axis andsubjective depth amounts D as the vertical axis. A unit of the Michelsoncontrast C is [%] and a unit of the subjective depth amount D is(arcmin: minute of angle).

In FIG. 2, squares ( ) represent the relation between Michelsoncontrasts C and the subjective depth amounts D in luminance imagesspatial frequencies of which are 0.5 cpd, 1 cpd and 2 cpd. Rhombusesrepresent the relation between Michelson contrasts C and the subjectivedepth amounts D in luminance images spatial frequencies of which are 4cpd and 8 cpd.

In the subjective depth amounts D, when a viewer feels that an objectdisplayed on the display exists on the display, the value is 0 [arcmin].It is shown that, when the subjective depth amount D is a positivevalue, the viewer feels that the object exists at a front position ofthe display by the value, and when the subject depth amount D is anegative value, the viewer feels that the object exists at a deepposition of the display.

For example, a plot of the square ( ) exists at a position where theMichelson contrast C is 5[%] and the subjective depth amount D isapproximately 0 [arcmin]. A plot of the square ( ) also exists at aposition where the Michelson contrast C is approximately 25[%] and thesubjective depth amount D is approximately 30 [arcmin]. The example ofthese values express that the subjective depth amount D of the viewer ischanged from approximately 0 [arcmin] to approximately 30 [arcmin] whenthe object the Michelson contrast C of which is 5[%] is changed to theobject the Michelson contrast C of which is approximately 25[%]. Thatis, it is shown that the viewer perceives that the object in the imageexist at a position approximately 30 [arcmin] nearer as compared withthe position before the change when the Michelson contrast C of theobject is changed from 5[%] to 25[%].

There is a monotone increasing relation between the Michelson contrast Cand the subjective depth amount D, in which the subjective depth amountD is increased as the Michelson contrast C is increased as shown in FIG.2. In other words, there is the relation in which the subjective depthamounts D are almost in proportion to logarithms of the Michelsoncontrasts C.

Accordingly, the relation between the Michelson contrast

C and the subjective depth amount D can be represented by performingapproximation with the performance function using logarithmsD=A×log(C)+B (A, B are constant numbers). Specifically, when parametersA, B of D=A×log(C)+B in which a residual will be the minimum arecalculated with respect to data of all spatial frequency components of0.5 cpd, 1 cpd, 2 cpd, 4 cpd and 8 cpd, they can be represented by anequation (4).

D=18.04×log(C)−29.07   (4)

That is, in the equation (4), A=18.04 and B=−29.07. In FIG. 2, theequation (4) is represented by a dotted line written as a “logfunction”.

The relation between the Michelson contrast C and the subjective depthamount D can be expressed by being applied to the Naka-Rushton equation(performance function) which is used with respect to the response of avisual system. The Naka-Rushton equation can be expressed by thefollowing equation (5).

$\begin{matrix}{D = {{D_{amp} \times \frac{C^{n}}{C^{n} + C_{50}^{n}}} + D_{\min}}} & (5)\end{matrix}$

In the equation (5), D_(amp), D_(min), C₅₀ and “n” are given constantnumbers, and D_(amp) represents the maximum minimum width of the depthamount, D_(min) represents the minimum value of the depth amount and C₅₀represents a contrast value obtained when the depth amount is at thecentral value between the maximum value and the minimum value.

When the parameters D_(amp), D_(min), C₅₀ and “n” of the Naka-Rushtonequation are calculated so that the residual will be the minimum withrespect to data of the all spatial frequencies of 0.5 cpd, 1 cpd, 2 cpd,4 cpd and 8 cpd obtained by the visual experiments, an equation (6) isobtained.

$\begin{matrix}{D = {{77.9 \times \frac{C^{1.09}}{C^{1.09} \times 7.74^{1.09}}} - 30.5}} & (6)\end{matrix}$

That is, in the equation (6), D_(amp)=77.9 m, D_(min)=−30.5 C₅₀=7.74 and“n”=1.09. In FIG. 2, the equation (6) represents by a chain line.

For example, in the case that the Michelson contrast C of a certainluminance image is 10[%], when C=10 is substituted in the equation (6),the subjective depth amount D is 13.9 [arcmin]. Then, in order to doublethe subjective depth amount D, namely, in order to make the value be27.8 [arcmin], reverse operation of the equation (6) is performed tomake the Michelson contrast C of the luminance image be 20.9[%].

The performance functions to be applied can be distinguished accordingto given spatial frequencies. For example, the Naka-Rushton equation isapplied as the performance function in FIG. 2, and when the performancefunctions are distinguished with respect to luminance images havingspatial frequencies of 2 cpd or less and with respect to luminanceimages having spatial frequencies higher than 2 cpd, D_(amp)=78.5,D_(min)=−30.5, C₅₀=9.93 and n=1.23 can be obtained with respect to dataof spatial frequency components of 2 cpd or less. Additionally,D_(amp)=58.5, D_(min)=−30.5, C₅₀=3.45 and n=2.05 can be obtained withrespect to data of spatial frequency components higher than 2 cpd.

That is, the performance function of an equation (7) can be applied toluminance images having spatial frequencies of 2 cpd or less, and theperformance function of an equation (8) can be applied to luminanceimages having spatial frequencies higher than 2 cpd.

$\begin{matrix}{D = {{78.5 \times \frac{C^{1.23}}{C^{1.23} \times 9.33^{1.23}}} - 30.5}} & (7) \\{D = {{58.5 \times \frac{C^{2.05}}{C^{2.05} + 3.45^{2.05}}} - 30.5}} & (8)\end{matrix}$

Return to FIG. 1, the memory unit 16 stores part or all of the equation(4), the equation (6) and a pair of the equation (7) and the equation(8) obtained by visual experiments. That is, all of the equation (4),the equation (6) and a pair of the equation (7) and the equation (8) arestored in the memory 16 and may be used according to need or, only theperformance functions determined to be applied in advance may be stored.When the calculation is performed by a computer, it is easy to performcalculation when using the equation (4) which is a logarithmic function.Additionally, the memory unit 16 may store equations of the performancefunctions directly as well as may store the performance functions in aform of a LUT (Look Up Table).

To the contrast adjustment unit 17, depth information of the 3D image issupplied from the depth information extraction unit 12, contrastcomponents of the 3D image are supplied from the contrast extractionunit 15 and spatial frequency components of the 3D image is suppliedfrom the spatial frequency extraction unit 14, respectively.

The contrast adjustment unit 17 changes (adjusts) the Michelsoncontrasts C of the luminance image to change (adjust) the depth sense ofthe 3D image. As explained with reference to FIG. 2, there is themonotone increasing relation between the Michelson contrast C and thesubjective depth amount D, therefore, the contrast components of theluminance image may be adjusted so that the Michelson contrast C isincreased when enhancing the depth sense of the 3D image.

A case in which the contrast adjustment unit 17 changes (adjusts) theMichelson contrasts C of the luminance image by using one performancefunction, for example, the performance function of the equation (6) withrespect to the whole luminance image regardless of the spatial frequencywill be explained.

For example, in the case that the Michelson contrast C of a certain oneprocessing unit region of the luminance image is 10[%], when C=10 issubstituted in the equation (6), the subjective depth amount D is 13.9[arcmin]. Then, in order to double the subjective depth amount D,namely, in order to make the value be 27.8 [arcmin], the contrastadjustment unit 17 may change (adjust) the Michelson contrast C in theprocessing unit region to 20.9[%], which is 2.09 times of the 10[%].

A depth adjustment amount for determining by what value the subjectivedepth amount D is multiplied is set in the contrast adjustment unit 17in advance. In order to double the subjective depth amount D, at leastone of a process of changing a region existing at a near side in theluminance image to be nearer and a process of changing a region existingat a deep side in the luminance image to be deeper is necessary. Thatis, the direction of changing the contrast differs in the near sideregion and in the deep side region. The depth information supplied fromthe depth information extraction unit 12 supplied from the depthinformation extraction unit 12 is used for separating the luminanceimage to the deep side region and the near side region.

The contrast adjustment unit 17 determines the deep side region and thenearside region of the 3D luminance image based on the depth informationsupplied from the depth information extraction unit 12. Here, theminimum unit of the region obtained by dividing the image into the deepside region and the near side region will be substantially equal to theprocessing unit region obtained by the contrast extraction unit 15 whichcalculates the Michelson contrasts C.

Next, the contrast adjustment unit 17 reads the performance function ofthe equation (6) stored in the memory unit 16. Then, the contrastadjustment unit 17 calculates the present subjective depth amounts Dconcerning all processing unit regions included in the deep side regionand the near side region from the Michelson contrasts C. The subjectivedepth amount D with respect to the processing region after adjusting thecontrast is determined from the calculated subjective depth amount D andthe depth adjustment amount which has been previously set, therefore,the Michelson contrast C to be set can be calculated based on theperformance function of the equation (6). That is, the contrastadjustment unit 17 calculates by what value the present Michelsoncontrast C is multiplied with respect to all processing unit regionsbased on the present subjective depth amount D and the depth informationof the image. Here, assume that the adjustment amount of the calculatedMichelson contrast C is M-times (M>0) , when the depth amount perceivedsubjectively from the adjusted image is intended to be larger than thepresent image, the adjustment value is M<1 in the deep side region andM>1 in the near side region. When the depth among perceivedsubstantially from the adjusted image is intended to be smaller than thepresent image, the adjustment value is M>1 in the deep side region andM<1 in the near side region.

In the above example, the present subjective depth amount D in the givenprocessing unit region in the image at the near side region iscalculated to be 13.9 [arcmin], and it is found that the Michelsoncontrast C is preferably approximately doubled (M=2.09) for changing thesubjective depth amount D to be doubled, namely, to be 27.8 [arcmin].

Next, the contrast adjustment unit 17 adjusts contrast components of the3D image so that the Michelson contrasts C are changed to values ofM-times as calculated with respect to all processing unit regionsincluded in the deep side region and the near side region.

Specifically, the contrast adjustment unit 17 performs Fourier transformin each processing unit region of the luminance image and calculatesspectrum intensity of frequency components included in each processingunit region. Then, the contrast adjustment unit 17 adjusts spectrumintensity of respective frequency components in each processing unitregion so that the Michelson contrasts C of each processing unit regionafter adjustment are changed to values of M-times. The spectrumintensity of respective frequency components in each processing unitregion after adjustment is supplied to the image processing unit 18.

The image processing unit 18 generates the luminance image obtainedafter adjusting contrast based on the adjustment result by the contrastadjustment unit 17. Specifically, the image processing unit 18 performsinverse Fourier transform to the spectrum intensity of respectivefrequency components in each processing unit region after adjustment tothereby calculate the luminance image obtained after adjusting eachprocessing unit region. Furthermore, the image processing unit 18generates the luminance image after adjusting contrast from theluminance image in respective processing unit regions after adjustment.That is, when respective processing unit regions are set so as to beoverlapped at parts thereof, plural luminance values after adjustmentmay be calculated with respect to each pixel of the luminance image.Accordingly, the image processing unit 18 generates the luminance imageafter adjusting contrast by applying an average value of pluralluminance values after adjustment as the luminance value of the pixelafter adjustment.

The image output unit 19 converts the luminance image generated by theimage processing unit 18 into the 3D image which is the same as theimage inputted to the image input unit 11, outputting the image to adevice of a subsequent stage (a display device and the like). In thecase of changing the luminance image represented by XYZ values in theCIE XYZ color system into the 3D image represented by RGB values in theRGB color system, conversion can be made by the following equations (9)to (11).

R=3.2410X−1.5374Y−0.4986Z   (9)

G=−0.9692X+1.8760Y+0.0416Z   (10)

B=0.0556X−0.2040Y+1.05702   (11)

When the data format at the time of outputting the 3D image isdesignated, the image output unit 19 may output the image afterconverting the data format of the 3D image into the designated format.

The image processing apparatus 1 of FIG. 1 is configured as describedabove.

In the above example, the Michelson contrast C of the nearside region inthe 3D luminance image is increased as well as the Michelson contrast Cof the deep side region is reduced to thereby enhance the deep sense.However, it is also possible to enhance the deep sense by increasingonly the Michelson contrast C of the near side region without changingthe

Michelson contrast C of the deep side region. Conversely, it is possibleto enhance the deep sense by reducing only the Michelson contrast C ofthe deep side region without changing the Michelson contrast C of thenear side region.

In the above example, the case in which the contrast adjustment unit 17adjusts Michelson contrasts C of the luminance image by using oneperformance function of the equation (6) with respect to the wholeluminance image has been explained, however, the case of making anadjustment by using the performance function of the equation (4) is thesame as the above.

Explanation of a Case in Which Plural Performance Functions are AppliedAccording to Given Spatial Frequency Components

Next, a case in which the contrast adjustment unit 17 adjusts theMichelson contrasts C of the luminance image by applying two performancefunctions of the equations (7) and the formula (8) will be explainedonly concerning a point different from the above case in which oneperformance function is applied.

The contrast adjustment unit 17 determines which spatial frequencycomponent is included at high rate in each processing unit region basedon spatial frequency components of the 3D image extracted by the spatialfrequency extraction unit 14.

When spatial frequency components of 2 cpd or less are included in theprocessing unit region at high rate, the contrast adjustment unit 17reads the performance function of the equation (7) from the memory unit16 and makes an adjustment of the Michelson contrast C in the samemanner as in the case of applying the performance function of theequation (6) with respect to the processing unit region.

On the other hand, when spatial frequency components larger than 2 cdare included at high rate in the processing unit region, the contrastadjustment unit 17 reads the performance function of the equation (8)from the memory unit 16 and makes an adjustment of the Michelsoncontrast C in the same manner as in the case of applying the performancefunction of the equation (6) with respect to the processing unit region.

As described above, when the adjustment of the Michelson contrast C ismade by applying plural performance functions, the performance functionto be applied is selected in accordance with the spatial frequencycomponents included in respective processing unit regions of theluminance image.

In the above example, the image is converted into the image havingdeeper depth to enhance the depth sense, however, it is naturallypossible to convert the 3D luminance image so as to reduce the depthsense.

Flowchart of Contrast Adjustment Processing

Next, contrast adjustment processing performed by the image processingapparatus 1 will be explained with reference to a flowchart of FIG. 3.On the explanation of respective steps of FIG. 3, explanation will bemade also with reference to FIG. 4 to FIG. 8 according to need.

First, in Step S11, the image input unit 11 receives input of a 3D imagefrom the outside and supplies the image to the depth informationextraction unit 12 and the luminance extraction unit 13. For example,the 3D image is inputted to the image input unit 11 from the outside inthe above first to third data formats.

FIG. 4 shows an example of the 3D image when supplied in the first dataformat. That is, in the first data format, a 3D image 31 including animage for a left eye 31L and an image for a right eye 31R is inputted tothe image input unit 11. The binocular parallax is set between the imagefor the left eye 31L and the image for the right eye 31R. The 3D image31 is the 3D image obtained by recording a situation in which a womanseats herself on a big stone against a background of an artificialwaterfall.

In Step S12, the depth information extraction unit 12 extracts depthinformation from the 3D image supplied from the image input unit 11. Asdescribed above, the depth information may be extracted from a 3Dluminance image obtained by extracting luminance components by theluminance extraction unit 13.

When data of the 3D image is supplied in the first or the second dataformat, the depth information extraction unit 12 calculates pixelmisalignment of corresponding points in stereoscopic images or viewpointimages, namely, the binocular parallax by using the methods such as theblock matching method and the DP matching method, and approximatelycalculates the depth information based on the calculated binocularparallax.

On the other hand, when the 3D image is supplied in the third dataformat, the depth information extraction unit 12 extracts the supplieddepth information as the depth information is directly supplied. Thedepth information extraction unit 12 supplies the extracted depthinformation to the contrast adjustment unit 17.

FIG. 5 shows a depth image 32 obtained by visualizing depth informationextracted with respect to the 3D image 31 including the image for theleft eye 31L and the image for the right eye 31R shown in FIG. 4.

In the depth image 32 of FIG. 5, the depth information is represented by8-bit values, and an object positioned at the deep side in the 3D image31 is represented by small pixels value and an object positioned at anear side is represented by large pixel values. The depth image 32 shownin FIG. 5 is for explaining extraction processing of depth informationin step S12 and the depth image 32 is not directly used in the contrastadjustment processing of FIG. 3.

When referring to the depth image 32 of FIG. 5, pixels of a woman andthe stone on which the woman sits are brighter than pixels of theartificial waterfall of the background (pixel values are larger).Therefore, the depth information accurately expresses that theartificial waterfall is the background and that the woman and the bigstone are the foreground.

In Step S13, the luminance extraction unit 13 extracts luminancecomponents of the 3D image supplied from the image input unit 11. Forexample, when the supplied 3D image is expressed by linear RGB values inthe RGB color system, the luminance extraction unit 13 extractsluminance components of the 3D image by converting the RGB values of the3D image into luminance values Y by using the above equation (1).

FIG. 6 shows a luminance image 33 including a luminance image for a lefteye 33L and a luminance image for a right eye 33R generated byextracting luminance components from the image for the left eye 31L andthe image for the right eye 31R shown in FIG. 4 respectively. In thedrawing, the difference between the 3D image 31 and the luminance image33 does not appear because of constraints thereof, however, the 3D mage31 is a color image and the luminance image 33 is a gray image.

In Step S14, the spatial frequency extraction unit 14 extracts givenspatial frequency components of the 3D image. In other words, thespatial frequency extraction unit 14 detects which spatial frequencycomponent is included in which region of the 3D image.

FIG. 7 shows spatial frequency component images obtained by visualizingthe extraction results at the time of extracting respective spatialfrequency components of 1 cpd, 2 cpd, 4 cpd and 8 cpd from the luminanceimage for the left eye 33L shown in FIG. 6.

The spatial frequency is defined by tint variation (contrast) of whiteand black entering into a viewing angle of 1 degree of human eyes asdescribed above, therefore, the spatial frequency depends on thedistance between the user who views the 3D image and the display and thesize of (the 3D image displayed on) the display. In the example, thespatial frequency is calculated under a condition that the user viewsthe display having the size of 40-inch and resolution of 1920×1080 atthe distance of 3 H (H is the display height).

A spatial frequency component image 41 of FIG. 7 is a spatial frequencycomponent image obtained by visualizing the extraction result when thespatial frequency component of 1 cpd was extracted with respect to theluminance image for the left eye 33L.

A spatial frequency component image 42 of FIG. 7 is a spatial frequencycomponent image obtained by visualizing the extraction result when thespatial frequency component of 2 cpd was extracted with respect to theluminance image for the left eye 33L.

A spatial frequency component image 43 of FIG. 7 is a spatial frequencycomponent image obtained by visualizing the extraction result when thespatial frequency component of 4 cpd was extracted with respect to theluminance image for the left eye 33L.

A spatial frequency component image 44 of FIG. 7 is a spatial frequencycomponent image obtained by visualizing the extraction result when thespatial frequency component of 8 cpd was extracted with respect to theluminance image for the left eye 33L.

In the spatial frequency component images 41 to 44, the higher theintensity of each spatial frequency component is, the brighter (higherthe pixel values) the pixels are.

The processing of Step S14 can be omitted when the Michelson contrasts Cof the luminance image are adjusted by using one function performancewith respect to the whole luminance image.

In Step S15, the contrast extraction unit 15 extracts contrastcomponents of the 3D image by using luminance components of the 3D imageextracted by the luminance extraction unit 13. Specifically, the thecontrast extraction unit 15 calculates the Michelson contrasts C ofrespective processing unit regions while moving the processing unitregion by a given amount with respect to the luminance image, therebycalculating the Michelson contrast C over the whole luminance image.

In Step S16, the contrast adjustment unit 17 determines a deep sideregion and a near side region based on depth information supplied fromthe deep information extraction unit 12.

In Step S17, the contrast adjustment unit 17 reads the performancefunction from the memory unit 16. When the contrast components areadjusted by applying one performance function with respect to the wholeluminance image, the performance function of the equation (4) or theequation (6) is read out. When the contrast components are adjusted byapplying two performance functions with respect to the whole luminanceimage, the performance functions of the equation (7) and the equation(8) are read out.

In Step S18, the contrast adjustment unit 17 calculates the presentsubjective depth amounts D from the Michelson contrasts C with respectto the whole processing unit regions. The subjective depth amounts Dwith respect to the processing regions after adjusting the contrast aredetermined by the calculated present subjective depth amounts D, theMichelson contrasts C which should be set can be calculated based on theperformance function read from the memory unit 16. That is, the contrastadjustment unit 17 calculates by what value the present Michelsoncontrast C is multiplied based on the present subjective depth amount Dand depth information of the image with respect to all processing unitregions. Assume that the adjustment value of the calculated Michelsoncontrast C is M-times.

Here, plural performance functions are applied according to givenspatial frequency components, the contrast adjustment unit 17 determineswhich spatial frequency component is included at high rate in theprocessing unit region of the luminance image based on the spatialfrequency components of the 3D image extracted in Step S14. Then, thecontrast adjustment unit 17 calculates the Michelson contrast C to beset in each processing unit region by using the performance functioncorresponding to the spatial frequency component determined at high ratein performance functions read from the memory unit 16 in Step S17.

On the other hand, when one performance function such as the equation(4) or the equation (6) is applied with respect to the whole luminanceimage, the contrast adjustment unit 17 calculates the Michelson contrastC to be set in each processing unit region by using one performancefunction read from the memory unit 16 in Step S17.

In Step S19, the contrast adjustment unit 17 adjusts the contrastcomponents C so that the Michelson contrasts C are changed to values ofM-times as calculated with respect to all processing unit regionsincluded in the deep side region and the near side region. Specifically,the contrast adjustment unit 17 performs Fourier transform in eachprocessing unit region of the luminance image and calculates frequencycomponents and spectrum intensity included in each processing unitregion. Then, the contrast adjustment unit 17 adjusts the spectrumintensity of respective frequency components in each processing unitregion so that the Michelson contrasts C of each processing unit regionafter adjustment are changed to values of M-times. The spectrumintensity of respective frequency components of each processing unitregion after adjustment is supplied to the image processing unit 18.

In Step S20, the image processing unit 18 generates the luminance imageafter adjusting contrast based on the adjustment result by the contrastadjustment unit 17. Specifically, the image processing unit 18 performsinverse Fourier transform to the spectrum intensity of respectivefrequency components in each processing unit region after adjustment tothereby calculate the luminance image obtained after adjusting eachprocessing unit region. Then, the image processing unit 18 generates theluminance image after adjusting contrast from the luminance imageobtained after adjusting each processing unit region.

In Step S21, the image output unit 19 outputs the 3D image obtainedafter adjusting contrast to a device of a subsequent stage such as adisplay device. That is, the image output unit converts the luminanceimage generated by the image processing unit 18 to the 3D image which isthe same as the image inputted in the pixel input unit 11 and outputsthe image to the subsequent device. The image output unit 19 may convertthe data format of the 3D image generated by the image processing unit18 to another format and output the image according to need.

The processing from Step S11 to Step S21 is executed repeatedly everytime the 3D image is inputted to the image input unit 11.

The order of executing respective processing of Steps S11 to S21 is notlimited to the above example and can be changed if necessary. Forexample, Step S12 and Step S13 can be executed in parallel in theprocessing of Steps S11 to S21. Step S14 and Step S15 can be alsoexecuted in parallel.

2. Second Embodiment Configuration Block Diagram of the Image ProcessingApparatus

FIG. 8 shows a configuration example of the image processing apparatusaccording to a second embodiment of the invention.

In FIG. 8, the same numerals are given to portions corresponding to thefirst embodiment and explanation thereof will be omitted.

The image processing apparatus 1 is configured in the same manner asFIG. 1 except that a user input unit 20 is newly provided.

In the second embodiment, the image processing result by Michelsoncontrasts C automatically adjusted (without input by the user) in thesame manner as the first embodiment is outputted at first. Then, afterconfirming the image processing result by the automatic adjustment, theuser himself/herself can change the Michelson contrasts C according toneed.

The user input unit 20 receives input of the Michelson contrast C as thedepth adjustment amount set by the user and supplies the value to thecontrast adjustment unit 17.

The contrast adjustment unit 17 allows the image output unit 19 tooutput an image for confirming whether the Michelson contrast C ischanged or not after the 3D image to which the contrast adjustmentprocessing has been performed based on the predetermined depthadjustment amount is outputted from the image output unit 19. Accordingto the processing, a confirmation screen for allowing the user toconfirm whether the Michelson contrast C is changed or not is displayedon the subsequent display device and the like. The confirmation screenmay be displayed by ODS on the 3D image to which the contrast adjustmentprocessing has been performed based on the automatic adjustment.

The user confirms the 3D image obtained after the contrast adjustmentprocessing based on the automatic adjustment and inputs a changed valueof the Michelson contrast C by the user input unit 20 when determiningto change the Michelson contrast C. The user input unit 20 supplies thechanged value of the Michelson contrast C inputted by the user to thecontrast adjustment unit 17.

When the changed value of the Michelson contrast C is supplied from theuser input unit 20, the contrast adjustment unit 17 applies the changedvalue of the Michelson contrast C in preference to the Michelsoncontrast C which has been automatically adjusted. That is, the contrastadjustment unit 17 adjusts contrast components of the 3D luminance imagebased on the changed value of the supplied Michelson contrast C, notbased on the adjustment amount of the Michelson contrast C based on thepredetermined depth adjustment amount.

Flowchart of Contrast Adjustment Processing

The contrast adjustment processing of the image processing apparatus 1according to the second embodiment will be explained with reference to aflowchart of FIG. 9.

Processing from Step S41 to Step S51 is the same as the above processingfrom Step S11 to S21 of FIG. 3, therefore, explanation thereof isomitted. That is, the 3D image in which luminance values have beenchanged by the Michelson contrasts C determined by the contrastadjustment unit 17 is displayed on the subsequent display device and thelike at first.

Then, in Step S52, the contrast adjustment unit 17 allows the imageoutput unit 19 to output an image to be confirmed whether the Michelsoncontrast C is changed or not and displays the confirmation screen forconfirming whether the Michelson contrast C is changed or not on thesubsequent display device and the like.

In Step S53, the contrast adjustment unit 17 determines whether thechange of the Michelson contrast C has been selected or not by the userinput unit 20. The selection information by the user concerning thepresence of change of the Michelson contrast C is supplied from the userinput unit 20 to the contrast adjustment unit 17.

When the change of the Michelson contrast C is selected, the changedvalue of the Michelson contrast C is also supplied from the user inputunit 20 to the contrast adjustment unit 17.

When the change of the Michelson contrast C is not selected in Step S53,the processing is ended.

On the other hand, when the change of the Michelson contrast C isselected in Step S53, the processing proceeds to Step S54. In Step S54,the user input unit 20 supplies the changed value of the Michelsoncontrast C which has been inputted by the user to the contrastadjustment unit 17.

After Step S54, the processing returns to Step S49. In Step S49, thecontrast adjustment unit 17 adjusts contrast components so that theMichelson contrasts C are changed to be changed values supplied from theuser input unit 20 with respect all processing unit regions included inthe deep side region and the near side region. The processing after StepS50 is the same.

According to the contrast adjustment processing as described above,after confirming the image processing result by the Michelson contrastsC which has been automatically adjusted, the user himself/herself canfurther change the Michelson contrast C according to need.

Accordingly, it is possible to adjust contrast components for enhancingthe depth sense of the 3D image in which user's taste is reflected.

It is also preferable to allow the user to select items such as “toincrease the depth sense” and “to reduce the depth sense” and to applythe amount previously set so as to correspond to the item (for example,1.5 times of the value determined by the automatic adjustment) as thechanged value of the Michelson contrast C.

According to the image processing apparatus 1 to which the invention isapplied, the depth sense of the 3D image can be enhanced by using theperformance function obtained by quantifying visual sensecharacteristics of human beings by visual experiments. Specifically, thedepth sense of the 3D image can be enhanced by using the performancefunction obtained by quantifying the relation between the Michelsoncontrast C and the depth amount D perceived subjectively by contrast.Contrast belongs to monocular 3D information in binocular 3D informationand monocular 3D information for allowing a human being to perceivestereoscopic effect and depth of an object. Therefore, it is possible toenhance the depth sense of the 3D image by adjusting parameters ofcontrast which is one of monocular 3D information based on visual sensecharacteristics of human beings according to the image processingapparatus 1.

Also according to the image processing apparatus 1, the depth sense ofthe 3D image can be enhanced by changing contrast, therefore, it ispossible to reduce binocular parallax which is one of binocular 3Dinformation due to the change. In the case, the binocular parallax issmall when comparing with a 3D image allowing the user to perceive thesame depth sense only by binocular parallax without changing contrast,therefore, “the convergence position and the adjustment position ofeyeballs” explained in the column of “Description of related art” comeclose to each other. Therefore, it is possible to reduce the viewer'sperception of artificiality, discomfort or visual fatigue. That is, the3D image with lesser sense of discomfort or uncomfortable feeling forhuman beings can be provided by adjusting parameters of contrast andenhancing the depth sense of the 3D image. In other words, it ispossible to provide the 3D image with lesser sense of discomfort oruncomfortable feeling for human beings by combining monocular 3Dinformation with binocular 3D information.

The above series of processing can be executed by hardware as well as bysoftware. When the series of processing is executed by software,programs included in the software are installed in a computer. Here, thecomputer includes a computer incorporated in dedicated hardware, ageneral-purpose personal computer which can execute various functions byinstalling various types of programs and the like.

FIG. 10 is a block diagram showing a configuration example of hardwareof a computer executing the above series of processing by programs.

In the computer, a CPU (Central Processing Unit) 101, a ROM (Read OnlyMemory) 102 and a RAM (Random Access Memory) 103 are mutually connectedby a bus 104.

An input/output interface 105 is further connected to the bus 104. Tothe input/output interface 105, an input unit 106, an output unit 107, astorage unit 108, a communication unit 109 and a drive 110 areconnected.

The input unit 106 includes a keyboard, a mouse, a microphone and thelike. The output unit 107 includes a display, a speaker and the like.The storage unit 108 includes a hard disk, a nonvolatile memory and thelike. The communication unit 109 includes a network interface and thelike. The drive 110 drives removal recording media 111 such as amagnetic disk, an optical disk, an magneto-optical disk or asemiconductor memory.

In the computer configured as described above, the CPU 101 loadsprograms stored in, for example, the storage unit 108 to the RAM 103through the input/output interface 105 and the bus 104 and executes theprograms, thereby performing the above series of processing.

The programs executed by the computer (CPU 101) can be provided by beingrecorded in the removal recording media 111 as packaged media. Theprograms can be also provided through wired or wireless transmissionmedia such as a local area network, Internet, digital satellitebroadcasting and so on.

In the computer, programs can be installed in the storage unit 108through the input/output interface 105 by mounting the removal recordingmedia 111 on the drive 110. Programs can be received by thecommunication unit 109 and installed in the storage unit 108 throughwired or wireless transmission media. Additionally, programs can bepreviously installed in the RAM 102 or the storage unit 108.

Programs executed by the computer may be programs processed in timeseries along the order explained by the present specification, or maybeprograms processed in parallel or processed at necessary timing such aswhen calling is performed.

The embodiment of the invention is not limited to the above embodimentsand can be variously changed within a scope not departing from the gistof the invention.

The present application contains subject matter related to thatdisclosed in Japanese Priority Patent Applications JP 2009-270077 filedin the Japan Patent Office on Nov. 27, 2009, the entire contents ofwhich is hereby incorporated by reference.

What is claimed is:
 1. An image processing apparatus comprising:circuitry configured to: receive an input image, the input image beingsupplied as a stereoscopic image including a left image for a left eyeand a right image for a right eye; calculate depth information for eachof a plurality of sub-regions of the input image based at least in parton the right image and the left image; and determine, for each of theplurality of sub-regions of the input image, at least one luminancecomponent based at least in part on the depth information and a functionindicating a relationship between depth information and luminance value.2. The image processing apparatus according to claim 1, wherein thecircuitry is further configured to: produce luminance adjusted right andleft images that comprise the at least one luminance component for eachof the plurality of sub-regions of the right image and the left image.3. The image processing apparatus according to claim 1, wherein thecircuitry is further configured to identify, based on the depthinformation, a first region of the input image as having a greater depththan a second region of the input image.
 4. The image processingapparatus according to claim 3, wherein the circuitry is furtherconfigured to perform at least one adjustment to luminance values of theinput image that is different for the first region of the input imageand the second region of the input image.
 5. The image processingapparatus according to claim 4, wherein the circuitry is furtherconfigured to perform an adjustment such that a luminance adjustmentdirection is different between the first region and the second region.6. The image processing apparatus according to claim 3, wherein thecircuitry is further configured to perform an adjustment at least on thefirst region or on the second region.
 7. The image processing apparatusaccording to claim 1, wherein the circuitry is further configured to:calculate one or more depth adjustment amounts from luminance componentsof the input image based at least in part on the function indicating arelationship between depth information and luminance value; and whereinthe determining, for each of the plurality of sub-regions of the rightimage and the left image, the at least one luminance component comprisesadjusting the luminance components of the input image based at least inpart on the one or more depth adjustment amounts.
 8. The imageprocessing apparatus according to claim 1, wherein the circuitry isfurther configured to receive an adjustment parameter, and to adjustluminance components of the input image based on the adjustmentparameter, the depth adjustment amount and the performance function. 9.The image processing apparatus according to claim 1, wherein thefunction is a performance function determined based at least in part onvisual sense characteristics of human beings.
 10. The image processingapparatus according to claim 1, wherein the function indicates arelationship between contrast and depth.
 11. The image processingapparatus according to claim 7, wherein the function indicates arelation between luminance components of an image and a depth amount.12. An image processing method comprising: receiving an input image, theinput image being supplied as a stereoscopic image including a leftimage for a left eye and a right image for a right eye; calculatingdepth information of each a plurality of sub-regions of the input imagebased on the right image and left image; and determining, for each ofthe plurality of sub-regions of the input image, at least one luminancecomponent based on the depth information and a function indicating arelationship between depth information and luminance value.
 13. Acomputer readable storage medium having instructions stored thereonthat, when executed by at least one processor perform a method, themethod comprising: receiving an input image, the input image beingsupplied as a stereoscopic image including a left image for a left eyeand a right image for a right eye; calculating a depth information ofeach a plurality of sub-regions of the input image based on the rightimage and left image; and determining, for each of the plurality ofsub-regions of the input image, at least one luminance component basedon the depth information and a function indicating a relationshipbetween depth information and luminance value.
 14. An image processingapparatus comprising: circuitry configured to: receive an input image;obtain depth information for each of a plurality of sub-regions of theinput image; and determine, for each of the plurality of sub-regions ofthe input image, at least one luminance component based at least in parton the depth information and a function indicating a relationshipbetween depth information and luminance value.
 15. The image processingapparatus according to claim 14, wherein the circuitry is furtherconfigured to extract the depth information from the input image. 16.The image processing apparatus according to claim 14, wherein the inputimage is supplied as a stereoscopic image including an left image for aleft eye and right image for a right eye.
 17. The image processingapparatus according to claim 14, wherein the input image is supplied asa multi-viewpoint image including three or more plural viewpoint images.18. The image processing apparatus according to claim 14, wherein theinput image is supplied as a two dimensional image and depth informationthereof.